Skip to content

chore(migration): Migrate code from googleapis/python-pubsub into packages/google-cloud-pubsub#16004

Draft
parthea wants to merge 1245 commits intomainfrom
migration.python-pubsub.migration.2026-03-02_16-59-45.migrate
Draft

chore(migration): Migrate code from googleapis/python-pubsub into packages/google-cloud-pubsub#16004
parthea wants to merge 1245 commits intomainfrom
migration.python-pubsub.migration.2026-03-02_16-59-45.migrate

Conversation

@parthea
Copy link
Contributor

@parthea parthea commented Mar 2, 2026

See #10930.

This PR should be merged with a merge-commit, not a squash-commit, in order to preserve the git history.

gcf-owl-bot bot and others added 30 commits June 27, 2023 13:08
Source-Link: googleapis/synthtool@909573c
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:ddf4551385d566771dc713090feb7b4c1164fb8a698fe52bbe7670b24236565b

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Source-Link: googleapis/synthtool@cb96037
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:2d816f26f728ac8b24248741e7d4c461c09764ef9f7be3684d557c9632e46dbd

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
…uffixes (#938)

- [ ] Regenerate this pull request now.

PiperOrigin-RevId: 543503563

Source-Link: https://togithub.com/googleapis/googleapis/commit/212ecef96e3ea7c67465ff3f52680301ad0ea1f9

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/532d27eea6ff128fdcbb3332176e894e76dcc685
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNTMyZDI3ZWVhNmZmMTI4ZmRjYmIzMzMyMTc2ZTg5NGU3NmRjYzY4NSJ9
* fix: Add async context manager return types

chore: Mock return_value should not populate oneof message fields

chore: Support snippet generation for services that only support REST transport

chore: Update gapic-generator-python to v1.11.0
PiperOrigin-RevId: 545430278

Source-Link: googleapis/googleapis@601b532

Source-Link: googleapis/googleapis-gen@b3f18d0
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYjNmMThkMGY2NTYwYTg1NTAyMmZkMDU4ODY1ZTc2MjA0NzlkN2FmOSJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
* Update README.rst

* Update README.rst

Co-authored-by: Anthonios Partheniou <partheniou@google.com>

---------

Co-authored-by: Anthonios Partheniou <partheniou@google.com>
* chore: Update gapic-generator-python to v1.11.2

PiperOrigin-RevId: 546510849

Source-Link: googleapis/googleapis@736073a

Source-Link: googleapis/googleapis-gen@deb64e8
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZGViNjRlOGVjMTlkMTQxZTMxMDg5ZmU5MzJiM2E5OTdhZDU0MWM0ZCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* Fix resource not found error for flaky test

---------

Co-authored-by: Anthonios Partheniou <partheniou@google.com>
)

Source-Link: googleapis/synthtool@d6103f4
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:39f0f3f2be02ef036e297e376fe3b6256775576da8a6ccb1d5eeb80f4c8bf8fb

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
* samples: Payload Unwrapping (NoWrapper)
* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* Update subscriber.py

---------

Co-authored-by: Anna Cocuzzo <63511057+acocuzzo@users.noreply.github.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Source-Link: googleapis/synthtool@eaef28e
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:f8ca7655fa8a449cadcabcbce4054f593dcbae7aeeab34aa3fcc8b5cf7a93c9e

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
Format string issue

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
- [ ] Make sure to open an issue as a [bug/issue](https://togithub.com/googleapis/python-pubsub/issues/new/choose) before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕
* chore: add flakyboy.yaml

* chore: add flakybot.yaml
…964)

Source-Link: googleapis/synthtool@395d53a
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:6c1cbc75c74b8bdd71dada2fa1677e9d6d78a889e9a70ee75b93d1d0543f96e1

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
…is set (#965)

* docs: clarified where ordering_key will be written if write_metadata is set

PiperOrigin-RevId: 551210991

Source-Link: googleapis/googleapis@7c762d7

Source-Link: googleapis/googleapis-gen@15fe4c5
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMTVmZTRjNWZmNWViZDUyOTExYjQyOWIwNWI5OTJlMjMyZjUzMzUxZSJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* add cloudstorage samples

* fix lint

* add protobuf

* Create unused topic
Source-Link: googleapis/synthtool@0ddbff8
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:bced5ca77c4dda0fd2f5d845d4035fc3c5d3d6b81f245246a36aee114970082b

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Source-Link: googleapis/synthtool@352b9d4
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:3e3800bb100af5d7f9e810d48212b37812c1856d20ffeafb99ebe66461b61fc7

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
…esour… (#971)

* fix: Change retry multiplier from 1.3 to 4, for requests that retry ResourceExhausted.
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
… RESOURCE_EXHAUSTD (#979)

* fix: Make retry policy back off more aggressively for RPCs that retry RESOURCE_EXHAUSTD

PiperOrigin-RevId: 557935020

Source-Link: googleapis/googleapis@38e1f31

Source-Link: googleapis/googleapis-gen@5bbe39d
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNWJiZTM5ZDM3MjE4YTQyMGIyMzY4YzM3NTQxZGU1ODg3ZGI3ZDZhZiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
abbrowne126 and others added 5 commits February 5, 2026 14:20
PR created by the Librarian CLI to initialize a release. Merging this PR
will auto trigger a release.

Librarian Version: v0.8.0
Language Image:
us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:b8058df4c45e9a6e07f6b4d65b458d0d059241dd34c814f151c8bf6b89211209
<details><summary>google-cloud-pubsub: 2.35.0</summary>

##
[2.35.0](googleapis/python-pubsub@v2.34.0...v2.35.0)
(2026-02-05)

### Features

* Add AIInference MessageTransform type (PiperOrigin-RevId: 853856321)
([07011139](googleapis/python-pubsub@07011139))

### Documentation

* A comment for field `analytics_hub_subscription_info` in message
`.google.pubsub.v1.Subscription` is updated (PiperOrigin-RevId:
853856321)
([07011139](googleapis/python-pubsub@07011139))

* Add the IDENTIFIER field behavior annotation to fields of Cloud
Pub/Sub methods that represent a specific identity and need to be
sourced with additional care (PiperOrigin-RevId: 840763233)
([07011139](googleapis/python-pubsub@07011139))

* A comment for field `topic` in message
`.google.pubsub.v1.Subscription` is updated (PiperOrigin-RevId:
853856321)
([07011139](googleapis/python-pubsub@07011139))

* A comment for field `subscription` in message
`.google.pubsub.v1.CreateSnapshotRequest` is updated (PiperOrigin-RevId:
853856321)
([07011139](googleapis/python-pubsub@07011139))

* add tags documentation links to Pub/Sub resource tags fields
(PiperOrigin-RevId: 845891076)
([07011139](googleapis/python-pubsub@07011139))

</details>
…1582)

This PR replaces the old yoshi-python team with cloud-sdk-python-team
and api-pubsub with pubsub-team.

b/478003109
…nto migration.python-pubsub.migration.2026-03-02_16-59-45.migrate
@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request completes a significant migration of the Google Cloud Pub/Sub Python client library's codebase into a dedicated packages/google-cloud-pubsub directory. This restructuring involves comprehensive updates to build, testing, and CI/CD configurations to support the new package layout. Additionally, it integrates OpenTelemetry tracing capabilities across both publishing and subscribing functionalities, enhancing observability. The changes also include various refinements to message handling, flow control, and sequencing mechanisms within the client library.

Highlights

  • Codebase Migration: The entire codebase for the Google Cloud Pub/Sub Python client library has been migrated into a new packages/google-cloud-pubsub directory, aligning with a monorepo structure.
  • Configuration Updates: Numerous configuration files, including .coveragerc, .flake8, .github workflows (for docs, lint, unittest), .kokoro CI/CD configurations, and .librarian generator inputs, have been added or updated to support the new package structure and development processes.
  • OpenTelemetry Tracing: New OpenTelemetry tracing support has been introduced for both publisher and subscriber clients, enabling context propagation and span management for message creation, flow control, batching, and subscriber operations (ack/nack/modack).
  • Publisher and Subscriber Enhancements: Refinements were made to publisher batching, flow control, and message sequencing logic, including handling of ordered messages and error conditions. Subscriber-side protocol elements like dispatcher, leaser, and streaming pull manager were also updated.
  • Documentation and Metadata: Project documentation (CHANGELOG.md, CONTRIBUTING.rst, README.rst, SECURITY.md, UPGRADING.md) and repository metadata (.repo-metadata.json) have been updated to reflect the new package location and current development guidelines.
Changelog
  • packages/google-cloud-pubsub/.librarian/generator-input/librarian.py
    • Updated Python script for library generation logic, including gRPC options, emulator support, deprecation warnings, and OpenTelemetry integration.
  • packages/google-cloud-pubsub/.librarian/generator-input/noxfile.py
    • Updated Nox configuration for testing, linting, and documentation generation.
  • packages/google-cloud-pubsub/.librarian/generator-input/setup.py
    • Updated setup.py configuration for package metadata and dependencies.
  • packages/google-cloud-pubsub/CHANGELOG.md
    • Updated the changelog to reflect recent releases and features.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/publisher/_batch/thread.py
    • Updated thread-based implementation for Pub/Sub message batching to include OpenTelemetry spans.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/publisher/_sequencer/ordered_sequencer.py
    • Updated ordered sequencer for Pub/Sub messages to pass OpenTelemetry data.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/publisher/_sequencer/unordered_sequencer.py
    • Updated unordered sequencer for Pub/Sub messages to pass OpenTelemetry data.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/publisher/client.py
    • Updated Publisher client implementation with OpenTelemetry tracing, batching, and flow control.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/publisher/flow_controller.py
    • Updated flow controller for managing message publishing limits.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py
    • Updated dispatcher for handling subscriber requests, including OpenTelemetry span management for ack/nack/modack.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/_protocol/leaser.py
    • Updated leaser for managing message leases, including OpenTelemetry data handling.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/_protocol/messages_on_hold.py
    • Updated mechanism for tracking messages on hold by ordering key to include OpenTelemetry spans.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/_protocol/requests.py
    • Updated named tuples for subscriber management requests to include OpenTelemetry data.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
    • Updated streaming pull manager for coordinating message consumption, including OpenTelemetry span management.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/client.py
    • Updated Subscriber client implementation with OpenTelemetry tracing, streaming pull, and flow control.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/message.py
    • Updated Message class for representing Pub/Sub messages, including OpenTelemetry data.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/scheduler.py
    • Updated scheduler for asynchronous callback execution.
  • packages/google-cloud-pubsub/google/cloud/pubsub_v1/types.py
    • Updated custom types and flow control settings for Pub/Sub clients, including OpenTelemetry tracing option.
  • packages/google-cloud-pubsub/google/pubsub/gapic_version.py
    • Updated GAPIC version to 2.35.0.
  • packages/google-cloud-pubsub/google/pubsub_v1/init.py
    • Updated __init__.py for pubsub_v1 package, including version checks and dependency warnings.
  • packages/google-cloud-pubsub/google/pubsub_v1/gapic_version.py
    • Updated GAPIC version to 2.35.0.
  • packages/google-cloud-pubsub/google/pubsub_v1/services/publisher/async_client.py
    • Updated asynchronous Publisher client.
Activity
  • This pull request is part of a larger migration effort, referencing issue Adopt split repo: _python-pubsub_ #10930.
  • The author has specified that this PR should be merged with a merge-commit to preserve git history.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request is a large migration that moves the google-cloud-pubsub package into this repository, adding a significant number of new configuration and source files. My review focused on these new files. I've identified a critical issue in the setup.py file concerning a non-existent dependency version which will break installation. Additionally, there are a few medium-severity issues related to outdated information in documentation (which is now linked to a rule about environment breaking changes), dead code, and a fragile implementation that relies on internal details of a standard library module. Addressing these points will improve the quality and robustness of the migrated package.

Note: Security Review is unavailable for this PR.


dependencies = [
"grpcio >= 1.51.3, < 2.0.0; python_version < '3.14'", # https://github.com/googleapis/python-pubsub/issues/609
"grpcio >= 1.75.1, < 2.0.0; python_version >= '3.14'",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The grpcio version 1.75.1 specified for python_version >= '3.14' does not appear to be a valid published version. This will likely cause installation failures for users on Python 3.14 and newer. Please verify and correct the required grpcio version.


> **WARNING**: Breaking change

The 2.0.0 release requires Python 3.6+.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This document, which is part of the current version of the library, states that the 2.0.0 release requires Python 3.6+. However, this library version requires Python 3.9+. This could be confusing for users. Please consider updating this to reflect the current minimum version requirement.

Suggested change
The 2.0.0 release requires Python 3.6+.
The 2.0.0 release requires Python 3.9+.
References
  1. When a release introduces breaking changes in environment requirements, such as dropping support for specific Python versions or major dependency versions (e.g., protobuf), prefer a minor version bump over a patch version bump. This maintains the ability to provide patches for the previous minor version for users still requiring the older environment.

Comment on lines +169 to +178
if (
self.publisher_options.enable_open_telemetry_tracing
and sys.version_info.major == 3
and sys.version_info.minor < 8
):
warnings.warn(
message="Open Telemetry for Python version 3.7 or lower is not supported. Disabling Open Telemetry tracing.",
category=RuntimeWarning,
)
self._open_telemetry_enabled = False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This version check for Python < 3.8 seems to be dead code, as the package requires Python 3.9 or newer according to setup.py. This block can be safely removed to improve code clarity.

Comment on lines +107 to +116
if (
self.subscriber_options.enable_open_telemetry_tracing
and sys.version_info.major == 3
and sys.version_info.minor < 8
):
warnings.warn(
message="Open Telemetry for Python version 3.7 or lower is not supported. Disabling Open Telemetry tracing.",
category=RuntimeWarning,
)
self._open_telemetry_enabled = False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similar to the publisher client, this version check for Python < 3.8 appears to be dead code since the library requires Python 3.9+. Removing this block would be a good cleanup.

Comment on lines +156 to +186
# Drop all pending item from the executor. Without this, the executor will also
# try to process any pending work items before termination, which is undesirable.
#
# TODO: Replace the logic below by passing `cancel_futures=True` to shutdown()
# once we only need to support Python 3.9+.
try:
while True:
work_item = self._executor._work_queue.get(block=False)
if work_item is None: # Exceutor in shutdown mode.
continue

dropped_message = None
if sys.version_info < (3, 14):
# For Python < 3.14, work_item.args is a tuple of positional arguments.
# The message is expected to be the first argument.
if hasattr(work_item, "args") and work_item.args:
dropped_message = work_item.args[0] # type: ignore[index]
else:
# For Python >= 3.14, work_item.task is (fn, args, kwargs).
# The message is expected to be the first item in the args tuple (task[1]).
if (
hasattr(work_item, "task")
and len(work_item.task) == 3
and work_item.task[1]
):
dropped_message = work_item.task[1][0]

if dropped_message is not None:
dropped_messages.append(dropped_message)
except queue.Empty:
pass
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This implementation relies on accessing the private _work_queue of ThreadPoolExecutor, which is fragile and could break in future Python versions. Since this library requires Python 3.9+, you could consider using the cancel_futures=True argument in executor.shutdown(), which was introduced in Python 3.9. To maintain the contract of returning dropped messages, you might need to maintain a separate collection of pending futures within the ThreadScheduler and, upon shutdown, cancel them and extract the message arguments from the futures that were not started.

@parthea parthea added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 2, 2026
@ohmayr ohmayr self-assigned this Mar 2, 2026
@ohmayr ohmayr force-pushed the migration.python-pubsub.migration.2026-03-02_16-59-45.migrate branch from efe1b30 to 5d47628 Compare March 4, 2026 04:01
@ohmayr ohmayr force-pushed the migration.python-pubsub.migration.2026-03-02_16-59-45.migrate branch from 4bb7b53 to 3ee28a6 Compare March 4, 2026 04:48
@ohmayr ohmayr force-pushed the migration.python-pubsub.migration.2026-03-02_16-59-45.migrate branch from 2bea99b to 6466a67 Compare March 5, 2026 09:33
@ohmayr ohmayr force-pushed the migration.python-pubsub.migration.2026-03-02_16-59-45.migrate branch from 6466a67 to c0f90fd Compare March 5, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not merge Indicates a pull request not ready for merge, due to either quality or timing.

Projects

None yet

Development

Successfully merging this pull request may close these issues.